Method and electronic device for identifying size of measurement target object

ABSTRACT

Provided are a method and electronic device for identifying a size of a measurement target object. The method includes imaging a reference object, which is a reference for identifying the size of the measurement target object, to acquire a reference object image, imaging the measurement target object to acquire a target object image, fusing the acquired reference object image and the acquired target object image, and inputting the fused reference object image and target object image to a first neural network model to acquire size information of the measurement target object from the first neural network model.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to and the benefit of Korean PatentApplication No. 2020-0119669, filed on Sep. 17, 2020, the disclosure ofwhich is incorporated herein by reference in its entirety.

BACKGROUND 1. Field of the Invention

The present disclosure relates to a device and method for identifying asize of a target object, and more particularly, to a device and methodfor identifying size of a target object using a neural network.

2. Discussion of Related Art

With the development of camera devices and display devices, vigorousresearch is underway on technologies for identifying the shape or sizeof an object in an image output through a display device. In particular,there are increasing attempts to reduce a defect rate and increase ayield of normal products by analyzing a product image through an imageanalysis technology in a mass product manufacturing process anddetecting a flaw of the product through the analysis result of theproduct image. Such an image analysis technology is used in variousindustrial fields such as product classification and inspection.

Generally, according to a related art, a distance sensor is used toestimate actual distance information from an image, or a referenceobject is imaged together with a measurement target object in order tomeasure size of the measurement target object. Also, it is necessary toacquire characteristic information (e.g., a focal length) of a cameraand the like in advance.

Therefore, it is necessary to develop a technology for accuratelyidentifying size of a measurement target product without imaging areference object together with the measurement target product oracquiring characteristic information of a camera in advance.

SUMMARY OF THE INVENTION

The present disclosure is directed to providing a method of identifyingsize of a measurement target object and an electronic device forperforming the method.

More specifically, the present disclosure is directed to providing amethod and device for identifying size of a measurement target object inan image using an artificial neural network.

According to an aspect of the present disclosure, there is provided amethod of identifying a size of a measurement target object by anelectronic device, the method including imaging a reference object,which is a reference for identifying the size of the measurement targetobject, to acquire a reference object image, imaging the measurementtarget object to acquire a target object image, fusing the acquiredreference object image and the acquired target object image, andinputting the fused reference object image and target object image to afirst neural network model to acquire size information of themeasurement target object from the first neural network model.

The method may further include inputting the target object image to asecond neural network model to acquire a target object mask imagegenerated on the basis of a target object area in the target objectimage from the second neural network model, and the fusing of theacquired reference object image and target object image may includefusing the target object mask image and the reference object image.

According to another aspect of the present disclosure, there is providedan electronic device for identifying a size of a measurement targetobject, the electronic device including a memory configured to store oneor more instructions and at least one processor configured to executethe one or more instructions. The at least one processor executes theone or more instructions so that a reference object, which is areference for identifying size of the measurement target object, isimaged to acquire a reference object image, the measurement targetobject is imaged to acquire a target object image, the acquiredreference object image and the acquired target object image are fused,and the fused reference object image and target object image is input toa first neural network model to acquire size information of themeasurement target object from the first neural network model.

According to another aspect of the present disclosure, there is provideda computer-readable recording medium in which a program for performing amethod of identifying a size of a measurement target object by anelectronic device is stored, wherein the method includes imaging areference object, which is a reference for identifying the size of themeasurement target object, to acquire a reference object image, imagingthe measurement target object to acquire a target object image, fusingthe acquired reference object image and the acquired target objectimage, and inputting the fused reference object image and target objectimage to a first neural network model to acquire size information of themeasurement target object from the first neural network model.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features and advantages of the presentdisclosure will become more apparent to those of ordinary skill in theart by describing exemplary embodiments thereof in detail with referenceto the accompanying drawings, in which:

FIG. 1 is a diagram schematically showing a process in which anelectronic device identifies size of a measurement target objectaccording to an exemplary embodiment;

FIG. 2 is a diagram showing a process in which an electronic deviceidentifies size of a measurement target object according to anotherexemplary embodiment;

FIG. 3 is a flowchart of a method in which an electronic deviceidentifies size of a measurement target object according to an exemplaryembodiment;

FIG. 4 is a diagram illustrating a first neural network model which isused by an electronic device to identify size of a target objectaccording to an exemplary embodiment;

FIG. 5 is a diagram illustrating a second neural network model which isused by an electronic device to identify size of a target objectaccording to an exemplary embodiment;

FIG. 6 is a diagram illustrating a process in which an electronic deviceidentifies size of a target object according to an exemplary embodiment;

FIG. 7 is a diagram illustrating a process in which an electronic devicefuses a target object image and a reference object image at an inputlevel according to an exemplary embodiment;

FIG. 8 is a flowchart of a method in which an electronic deviceidentifies size of a target object according to another exemplaryembodiment;

FIG. 9 is a diagram illustrating a process in which an electronic deviceidentifies size of a target object according to another exemplaryembodiment;

FIG. 10 is a flowchart of a method in which an electronic deviceidentifies size of a target object according to another exemplaryembodiment;

FIG. 11 is a diagram showing a process in which an electronic devicefuses an output value of a second neural network model, which is outputwhen a target object image is input to the second neural network model,and an output value of the second neural network model, which is outputwhen a reference object image is input to the second neural networkmodel, at an output value level of a neural network model according toanother exemplary embodiment;

FIG. 12 is a flowchart of a method in which an electronic deviceidentifies size of a target object according to another exemplaryembodiment;

FIG. 13 is a diagram showing a process in which an electronic devicefuses a target object feature and a reference object feature on thebasis of feature units according to another exemplary embodiment;

FIG. 14 is a diagram illustrating a process in which an electronicdevice measures size of a target object using a first neural networkaccording to an exemplary embodiment;

FIG. 15 is a diagram illustrating training data which is generated by anelectronic device to train a neural network model according to anexemplary embodiment;

FIG. 16 is a block diagram of an electronic device according to anexemplary embodiment; and

FIG. 17 is a block diagram of an electronic device according to anotherexemplary embodiment.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

Terminology used herein will be briefly described, and then the presentdisclosure will be described in detail.

Although general terms widely used at present are selected inconsideration of the functions in the present disclosure, these termsmay vary according to intentions of those of ordinary skill in the art,precedents, the advent of new technologies, or the like. Termsarbitrarily selected by the applicant may be used in a specific case. Inthis case, their meanings will be given in the detailed description ofthe present disclosure. Accordingly, the terms used in the presentdisclosure should not be simply defined on the basis of their names butdefined on the basis of their meanings and the content of the entirespecification.

Throughout the specification, when a part is referred to as “including”an element, the part may additionally include another element and doesnot preclude another element unless particularly stated otherwise. Also,a term, such as “ . . . unit” or “module,” used herein represents a unitfor processing at least one function or operation and may be implementedas hardware, software, or a combination of hardware and software.

Hereinafter, exemplary embodiments of the present disclosure will bedescribed in detail with reference to the accompanying drawings so thatthe exemplary embodiments can be easily implemented by those of ordinaryskill in the art to which the present disclosure pertains. However, thepresent disclosure may be embodied in many different forms and is notlimited to the exemplary embodiments set forth herein. In the drawings,parts irrelevant to the description are omitted to clearly describe thepresent disclosure. Throughout the specification, like referencenumerals refer to like elements.

FIG. 1 is a diagram schematically showing a process in which anelectronic device identifies size of a measurement target objectaccording to an exemplary embodiment.

According to an exemplary embodiment, an electronic device 1000 mayacquire a reference object image 112 and a target object image 114 froman external device 3000 and analyze the acquired reference object image112 and target object image 114 to identify a size 134 of a measurementtarget object in the target object image 114. According to an exemplaryembodiment, the electronic device 1000 may identify the size 134 of themeasurement target object as one piece of measurement target objectinformation 132 by analyzing the target object image 114. According toan exemplary embodiment, the external device 3000 may be a camera devicefor imaging a reference object or the measurement target object.

Although FIG. 1 shows that the electronic device 1000 receives thereference object image 112 and the target object image 114 from theexternal device 3000, according to another exemplary embodiment, theelectronic device 1000 may include at least one camera and directlyacquire the reference object image 112 and the target object image 114.According to an exemplary embodiment, the reference object image 112 maybe an image generated by imaging a reference object, and the targetobject image 114 may be an image generated by imaging a measurementtarget object. According to an exemplary embodiment, the target objectimage 114 may include an image of at least one measurement targetobject.

According to an exemplary embodiment, the electronic device 1000 mayinclude at least one neural network model. According to an exemplaryembodiment, the electronic device 1000 may include a first neuralnetwork model 122 and a second neural network model 124. The electronicdevice 1000 may analyze the reference object image 112 and the targetobject image 114 using at least one of the above-described neuralnetwork models, thereby identifying the size 134 of the measurementtarget object in the target object image 114.

Although FIG. 1 shows that the electronic device 1000 identifies size ofa measurement target object, according to another exemplary embodiment,the electronic device 1000 may identify not only size of a measurementtarget object but also the type of measurement target object and anobject area in which the measurement target object is present in thetarget object image 114.

According to an exemplary embodiment, the electronic device 1000 may beimplemented in various forms. For example, the electronic device 1000described herein may be a digital camera, a mobile terminal, a smartphone, a laptop computer, a tablet personal computer (PC), an e-bookterminal, a digital broadcast terminal, a personal digital assistant(PDA), a portable multimedia player (PMP), a navigation device, a MovingPicture Experts Group (MPEG) Audio Layer III (MP3) player, or the like,but the electronic device 1000 is not limited thereto.

According to an exemplary embodiment, the external device 3000 mayinclude a camera 102, a camera driving device 104, and a networkinterface 106. However, the external device 3000 is not limited theretoand may further include other devices for capturing an image of anobject and transmitting information on the captured image to theelectronic device 1000. According to an exemplary embodiment, theexternal device 3000 may be a camera device, a still image capturingdevice, or a video capturing device including at least one camera.

According to an exemplary embodiment, the camera 102 may include atleast one image sensor and generate an image by sensing light reflectedby an object through the image sensor. The camera driving device 104 maycontrol the position or attitude of the camera 102. The networkinterface 106 may transmit, to the electronic device 1000, informationon the object that the external device 300 acquires by imaging theobject.

According to an exemplary embodiment, the electronic device 1000 mayidentify size of a measurement target object in a target object image byinteroperating with a server 2000. According to an exemplary embodiment,the server 2000 may be connected to the external device 3000 to acquirethe reference object image 112 and the target object image 114 andtransmit information on the acquired reference object image 112 andtarget object image 114 to the electronic device 1000.

According to an exemplary embodiment, the server 2000 may include othercomputing devices that may communicate with the electronic device 1000,which measures size of a measurement target object, through a network toexchange data with the electronic device 1000. According to an exemplaryembodiment, the network includes a local area network (LAN), a wide areanetwork (WAN), a value added network (VAN), a mobile radio communicationnetwork, a satellite communication network, and combinations thereof.The network is a comprehensive data communication network which allowsthe network components shown in FIG. 1 to smoothly communicate with eachother and may include a wired Internet, a wireless Internet, and amobile wireless communication network.

FIG. 2 is a diagram showing a process in which an electronic deviceidentifies size of a measurement target object according to anotherexemplary embodiment.

Referring to FIG. 2, a process in which the electronic device 1000identifies size of a measurement target object will be described.According to an exemplary embodiment, the electronic device 1000 may beconnected to a first camera 202 or a second camera 206 which is fixed ata different attitude than the first camera 202. According to anexemplary embodiment, the first camera 202 and the second camera 206 maybe set to different attitudes at different positions. Also, according toan exemplary embodiment, the first camera 202 and the second camera 206may be different cameras or may be the same camera that is differentonly in attitude.

According to an exemplary embodiment, the electronic device 1000 mayacquire a first reference image 212 from the first camera 202 and inputthe acquired first reference image 212 to a second neural network model222. The electronic device 1000 may acquire a first reference objectfeature 224 corresponding to the first reference image 212 from thesecond neural network model 222 and input the acquired first referenceobject feature 224 to a first neural network model 228.

According to an exemplary embodiment, the electronic device 1000 mayinput a first target object image 232, which is acquired by imaging afirst measurement target object through a first camera 204, to the firstneural network model 228. The electronic device 1000 may input the firsttarget object image 232 to the first neural network model 228 togetherwith the first reference object feature 224. The electronic device 1000may acquire information on a first actual size 236 of the measurementtarget object in the first target object image 232 from the first neuralnetwork model 228. According to an exemplary embodiment, the firstcamera 202 may be a camera which has the same attitude at the sameposition as the first camera 204.

According to another exemplary embodiment, the electronic device 1000may acquire a second reference image 214 from the second camera 206 andinput the acquired second reference image 214 to the second neuralnetwork model 222. The electronic device 1000 may acquire a secondreference object feature 226 corresponding to the second reference image214 from the second neural network model 222 and input the acquiredsecond reference object feature 226 to the first neural network model228.

According to an exemplary embodiment, the electronic device 1000 mayinput a second target object image 234, which is acquired by imaging asecond measurement target object through a second camera 208, to thefirst neural network model 228. The electronic device 1000 may input thesecond target object image 234 to the first neural network model 228together with the second reference object feature 226. The electronicdevice 1000 may acquire information on a second actual size 238 of themeasurement target object in the second target object image 234 from thefirst neural network model 228.

According to an exemplary embodiment, the second camera 206 may be acamera which has the same attitude at the same position as the secondcamera 208.

FIG. 3 is a flowchart of a method in which an electronic deviceidentifies size of a measurement target object according to an exemplaryembodiment.

In operation S310, the electronic device 1000 may acquire a referenceobject image by imaging a reference object which is a reference foridentifying size of a measurement target object. According to anexemplary embodiment, the electronic device 1000 may receive thereference object image, which is acquired by imaging the referenceobject, from a camera device connected thereto.

In operation S320, the electronic device 1000 may acquire a targetobject image by imaging the measurement target object. According to anexemplary embodiment, the electronic device 1000 may acquire the targetobject image, which is acquired by imaging the measurement targetobject, from a camera connected to thereto.

In operation S330, the electronic device 1000 may fuse the referenceobject image and the target object image. For example, the electronicdevice 1000 may fuse the reference object image and the target objectimage by applying image information of the reference object shown in thereference object image to the target object image. According to anotherexemplary embodiment, the electronic device 1000 may acquire a targetobject mask image which may be generated by inputting the target objectimage to a second neural network model and fuse the acquired targetobject mask image and the reference object image.

According to an exemplary embodiment, the electronic device 1000 mayfuse the reference object image and the target object image by applyingthe reference object image information to the target object image or mayfuse the reference object image and the target object image in units ofcertain channels.

In operation S340, the electronic device 1000 may acquire size of themeasurement target object from a first neural network model by inputtingthe fused reference object image and target object image to the firstneural network model. According to another exemplary embodiment, theelectronic device 1000 may acquire size information of the measurementtarget object by inputting the fused target object mask image andreference object image to the first neural network model.

According to an exemplary embodiment, the first neural network model andthe second neural network model used by the electronic device 1000 mayinclude a deep neural network (DNN). For example, the first neuralnetwork model and the second neural network model may be convolutionalneural networks (CNNs), DNNs, recurrent neural networks (RNNs),restricted Boltzmann machines (RBMs), deep belief networks (DBNs),bidirectional recurrent deep neural networks (BRDNNs), deep Q-networks,or the like but are not limited thereto.

FIG. 4 is a diagram illustrating a first neural network model which isused by an electronic device to identify size of a target objectaccording to an exemplary embodiment.

According to an exemplary embodiment, the electronic device 1000 mayacquire size information of a measurement target object in a targetobject image 402 as an identification result 408 using a first neuralnetwork model 406. For example, the electronic device 1000 may acquirethe target object image 402 by imaging the measurement target object andacquire a reference object image 404 by imaging a reference object. Theelectronic device 1000 may input the target object image 402 and thereference object image 404 to the first neural network model 406.

According to another exemplary embodiment, the electronic device 1000may acquire a target object mask image, which is a binary image, on thebasis of an object area in the target object image 402 by inputting thetarget object image 402 to the second neural network model and theninput the target object mask image and the reference object image 404 tothe first neural network model 406.

According to an exemplary embodiment, the first neural network model 406is a DNN and may be a neural network having a ResNet-32 structure.However, the first neural network model 406 is not limited thereto.According to an exemplary embodiment, the first neural network model 406may be trained in advance to output information of a measurement targetobject in a target object image when the target object image and areference object image are input. According to an exemplary embodiment,the first neural network model 406 may include an object sizeidentification network which acquires the target object mask image andthe reference object image 404, compares the acquired target object maskimage and reference object image 404, and outputs information on thetype and size of the measurement target object, which is represented byan object box in an image including the measurement target object, onthe basis of the comparison result.

According to an exemplary embodiment, the first neural network model 406may further output information on a type of the measurement targetobject and an area, which is represented by the measurement targetobject in the target object image 402, in addition to the size of themeasurement target object. The electronic device 1000 may acquireinformation on the size, type, or area of the measurement target objectas an identification result in the target object image 402 on the basisof the output value of the first neural network model 406.

FIG. 5 is a diagram illustrating a second neural network model which isused by an electronic device to identify size of a target objectaccording to an exemplary embodiment.

According to an exemplary embodiment, the electronic device 1000 mayacquire at least one target object mask image using a second neuralnetwork model 504. For example, the electronic device 1000 may acquireone or more target object mask images 506 and 508 by acquiring a targetobject image 502 and inputting the acquired target object image 502 tothe second neural network model 504. According to an exemplaryembodiment, the second neural network model 504 may be a mask regionbased (R)-CNN but is not limited thereto.

According to an exemplary embodiment, the second neural network model504 may include an object detection network. The object detectionnetwork identifies an object area including each of one or moremeasurement target objects in the target object image 502, which isinput to the second neural network model 504 and includes the at leastmeasurement target object, generates an object box including the objectarea, identifies the type of measurement target object represented bythe object box, and binarizes an image input to the second neuralnetwork model 504 on the basis of the object area to generate the targetobject mask image.

According to an exemplary embodiment, by using the second neural networkmodel 504, the electronic device 1000 may identify the boundary of anobject area on the basis of pixel values in the target object image 502,identify an object area on the basis of the identified boundary, andbinarize the identified object area and the target object imageexcluding the object area to generate a target object mask image. Theelectronic device 1000 may acquire size information of the measurementtarget object by fusing the generated target object mask image and areference object image and inputting the fused image to a first neuralnetwork model.

FIG. 6 is a diagram illustrating a process in which an electronic deviceidentifies size of a target object according to an exemplary embodiment.

Referring to FIG. 6, a process in which the electronic device 1000identifies size of a target object will be described in brief. Theelectronic device 1000 may acquire a reference object image 602 and atarget object image 604 and fuse the acquired reference object image 602and target object image 604 to generate a fusion image 606. According toanother exemplary embodiment, the electronic device 1000 may fuse atarget object mask image generated by masking a target object image anda reference object image.

The electronic device 1000 may input the fusion image 606 to a firstneural network model 609. According to an exemplary embodiment, theelectronic device 1000 may transfer the fusion image 606 to an objectsize identification network 608 in the first neural network model 609.The electronic device 1000 may identify a size 612 of a target object inthe target object image 604 on the basis of an output value of the firstneural network model 609. According to an exemplary embodiment, theelectronic device 1000 may include an object size identification unit607. Using the object size identification unit 607, the electronicdevice 1000 may fuse the reference object image 602 and the targetobject image 604 and input the fusion image 606 generated as a result offusion to the object size identification network 608.

FIG. 7 is a diagram illustrating a process in which an electronic devicefuses a target object image and a reference object image at an inputlevel according to an exemplary embodiment.

The electronic device 1000 according to the present disclosure may fuseinformation on a target object image and information on a referenceobject image at various levels. According to an exemplary embodiment,referring to FIG. 7, the electronic device 1000 may fuse a target objectimage 702 and a reference object image 704 at an image level before thetarget object image 702 and the reference object image 704 are input toa first neural network model 708. The electronic device 1000 may acquiresize information of a measurement target object as an identificationresult 712 by inputting the fused target object image 702 and referenceobject image 704 to the first neural network model 708.

According to another exemplary embodiment, the electronic device 1000may fuse a target object mask image, which is acquired from a secondneural network model by inputting the target object image 702 to thesecond neural network model, with the reference object image 704 andinput a fusion image generated as a result of fusion to the first neuralnetwork model 708.

FIG. 8 is a flowchart of a method in which an electronic deviceidentifies size of a target object according to another exemplaryembodiment.

In operation S810, the electronic device 1000 may acquire a referenceobject image by imaging a reference object which is a reference foridentifying size of a measurement target object. Since operation S810may correspond to operation S310 of FIG. 3, a detailed descriptionthereof is omitted. In operation S820, the electronic device 1000 mayacquire a target object image by imaging the measurement target object.Since operation S820 may correspond to operation S320 of FIG. 3, adetailed description thereof is omitted.

In operation S830, the electronic device 1000 may acquire a referenceobject feature from a second neural network model by inputting thereference object image to the second neural network model. According toan exemplary embodiment, the reference object feature is an output valueof a network layer in the second neural network model and may be avector sequence of a specific layer. According to an exemplaryembodiment, the reference object feature is an output value of thenetwork layer in the second neural network model and may be a vectorsequence of hidden layer vectors.

In operation S840, the electronic device 1000 may acquire sizeinformation of the measurement target object by inputting the referenceobject feature and the target object image to a first neural networkmodel.

FIG. 9 is a diagram illustrating a process in which an electronic deviceidentifies size of a target object according to another exemplaryembodiment.

Referring to FIG. 9, another process in which the electronic device 1000identifies size of a target object will be described in brief. Theelectronic device 1000 may acquire a reference object image 902 and atarget object image 904 and acquire a reference object feature 908 byinputting the acquired reference object image 902 to a second neuralnetwork model 906. The electronic device 1000 may acquire information ona target object size 914 by inputting the reference object feature 908and the target object image 904 to a first neural network model 912.According to an exemplary embodiment, an operation in which theelectronic device 1000 inputs the reference object image 902 and thetarget object image 904 to the second neural network model 906, acquiresthe reference object feature 908 from the second neural network model906, and inputs the acquired reference object feature 908 and the targetobject image 904 to the first neural network model 912 may be performedby a size identification unit 905 in the electronic device 1000.

FIG. 10 is a flowchart of a method in which an electronic deviceidentifies size of a target object according to another exemplaryembodiment.

In operation S1010, the electronic device 1000 may acquire a referenceobject image by imaging a reference object which is a reference foridentifying size of a measurement target object. Since operation S1010may correspond to operation S310 of FIG. 3, a detailed descriptionthereof is omitted. In operation S1020, the electronic device 1000 mayacquire a target object image by imaging the measurement target object.Since operation S1020 may correspond to operation S320 of FIG. 3, adetailed description thereof is omitted.

In operation S1030, the electronic device 1000 may acquire a firstoutput value of a second neural network model by inputting the referenceobject image to the second neural network model. According to anexemplary embodiment, the electronic device 1000 may acquire a vectorsequence, which is output from an output layer of the second neuralnetwork model when the reference object image is input to the secondneural network model, as the first output value of the second neuralnetwork model. According to another exemplary embodiment, the electronicdevice 1000 may acquire a vector sequence, which is output from onelayer selected in the second neural network model when the referenceobject image is input to the second neural network model, as the firstoutput value of the second neural network model.

In operation S1040, the electronic device 1000 may acquire a secondoutput value of the second neural network model by inputting the targetobject image to the second neural network model. According to anexemplary embodiment, the electronic device 1000 may acquire a vectorsequence, which is output from the output layer of the second neuralnetwork model when the target object image is input to the second neuralnetwork model, as the second output value of the second neural networkmodel. According to another exemplary embodiment, the electronic device1000 may acquire a vector sequence, which is output from one layerselected in the second neural network model when the target object imageis input to the second neural network model, as the second output valueof the second neural network model.

In operation S1050, the electronic device 1000 may fuse the first outputvalue and the second output value of the second neural network model.According to an exemplary embodiment, the electronic device 1000 mayfuse the first output value and the second output value of the secondneural network model by averaging the first output value and the secondvalue. According to another exemplary embodiment, when each of the firstoutput value and the second output value is an output value of a Softmaxlayer in the second neural network model, the electronic device 1000 mayfuse the first output value and the second output value by averaging theoutput values of the Softmax layer.

In operation S1060, the electronic device 1000 may acquire sizeinformation of the measurement target object by inputting the fusedfirst output value and second output value of the second neural networkmodel to a first neural network model.

FIG. 11 is a diagram showing a process in which an electronic devicefuses an output value of a second neural network model, which is outputwhen a target object image is input to the second neural network model,and an output value of the second neural network model, which is outputwhen a reference object image is input to the second neural networkmodel, at an output value level of a neural network model according toanother exemplary embodiment.

The process in which the electronic device 1000 identifies size of atarget object described above with reference to FIG. 10 will bedescribed in brief with reference to FIG. 11. According to an exemplaryembodiment, the electronic device 1000 may acquire a first output valueand a second output value of a second neural network model 1106respectively corresponding to a target object image 1102 and a referenceobject image 1104 by inputting the target object image 1102 and thereference object image 1104 to the second neural network model 1106.

The electronic device 1000 may acquire a fused output value by fusingthe above-described first and second output values of the second neuralnetwork model at an output unit level of the second neural network modeland acquire size information of a measurement target object as anidentification result 1112 by inputting the acquired fused output valueto a first neural network model 1108.

The electronic device 1000 according to the present disclosure may fusethe first output value and the second output value of the second neuralnetwork model respectively corresponding to the target object image 1102and the reference object image 1104 by executing an output fusionalgorithm at an output layer level of the second neural network model.

FIG. 12 is a flowchart of a method in which an electronic deviceidentifies size of a target object according to another exemplaryembodiment. In operation S1210, the electronic device 1000 may acquire areference object image by imaging a reference object which is areference for identifying size of a measurement target object. Sinceoperation S1210 may correspond to operation S310 of FIG. 3, a detaileddescription thereof is omitted. In operation S1220, the electronicdevice 1000 may acquire a target object image by imaging the measurementtarget object. Since operation S1220 may correspond to operation S320 ofFIG. 3, a detailed description thereof is omitted.

In operation S1230, the electronic device 1000 may acquire a referenceobject feature from a second neural network model by inputting thereference object image to the second neural network model. For example,the electronic device 1000 may acquire a vector sequence output from atleast one layer in the second neural network model as a reference objectfeature by inputting the reference object image to the second neuralnetwork model. According to an exemplary embodiment, the referenceobject feature may be generated in the form of a vector including acertain sequence.

In operation S1240, the electronic device 1000 may acquire a targetobject feature from the second neural network model by inputting thetarget object image to the second neural network model. For example, theelectronic device 1000 may acquire a vector sequence output from atleast one layer in the second neural network model as a target objectfeature by inputting the target object image to the second neuralnetwork model. According to an exemplary embodiment, the target objectfeature may be generated in the form of a vector including a certainsequence.

In operation S1250, the electronic device 1000 may fuse the acquiredreference object feature and target object feature. For example, theelectronic device 1000 may fuse the reference object feature and thetarget object feature by performing element-wise addition of thereference object feature and the target object feature. According toanother exemplary embodiment, the electronic device 1000 may fuse theacquired reference object feature and target object feature byconcatenating the reference object feature and the target object featurein units of channels.

In operation S1260, the electronic device 1000 may acquire sizeinformation of the measurement target object by inputting the fusedreference object feature and target object feature to a first neuralnetwork model.

FIG. 13 is a diagram showing a process in which an electronic devicefuses a target object feature and a reference object feature on thebasis of feature units according to another exemplary embodiment.

The process of FIG. 12 in which the electronic device 1000 identifiessize of a measurement target object will be described with reference toFIG. 13 focusing on a fusion operation. For example, the electronicdevice 1000 may acquire a target object image 1302 and a referenceobject image 1304 and input the acquired target object image 1302 andreference object image 1304 to a second neural network model 1306 toacquire a target object feature and a reference object feature, each ofwhich is output from the second neural network model 1306.

The electronic device 1000 according to the present disclosure may fusethe target object feature and the reference object feature output fromthe second neural network model by executing a feature fusion algorithmfor fusing the target object feature and the reference object feature ata feature level. The feature fusion algorithm executed by the electronicdevice 1000 may include an algorithm for the above-describedelement-wise addition or concatenation.

The electronic device 1000 may acquire size information of a measurementtarget object as a result 1312 by inputting the target object featureand the reference object feature fused at the feature level to a firstneural network model.

As described above, to measure size of a measurement target object in atarget object image, the electronic device 1000 according to the presentdisclosure may fuse reference object information and target objectinformation at various levels. The electronic device 1000 according tothe present disclosure may accurately identify size of a target objectimage by inputting a result of at least one of image fusion at an imagelevel, image fusion at an output level of a neural network layer, andfusion at a level of features output from a neural network layer to afirst neural network model.

FIG. 14 is a diagram illustrating a process in which an electronicdevice measures size of a target object using a first neural networkaccording to an exemplary embodiment.

According to an exemplary embodiment, the electronic device 1000 maycompare a first reference object 1404 with each target object imageusing a first neural network model. For example, the electronic device1000 may acquire a reference object image by imaging the first referenceobject 1404 through a first camera 1402 and determine a first referenceobject feature 1406, a second reference object feature 1408, etc. bycomparing the acquired reference object image with each target objectimage including various measurement target objects.

FIG. 15 is a diagram illustrating training data which is generated by anelectronic device to train a neural network model according to anexemplary embodiment.

Referring to FIG. 15, training data 1514 which is used by the electronicdevice 1000 to train a first neural network model and a second neuralnetwork model is shown.

According to an exemplary embodiment, the electronic device 1000 mayacquire a preset three-dimensional (3D) computer aided design (CAD)model 1511 and a camera characteristic model 1510 including a presetcamera characteristic parameter in a virtual environment in which ameasurement target object and a reference object may be virtually imagedand image various types of measurement target objects 1512 shown in the3D CAD model 1511 through the camera characteristic model 1510 togenerate training data 1514. According to an exemplary embodiment, theelectronic device 1000 may image the various types of measurement targetobjects 1512 shown in the 3D CAD model 1511 so that the training data1514 may be generated to further reflect an object image which is abase.

While a surrounding environment (light intensity, lighting, etc.) and anaperture change are all taken into consideration in the case ofcollecting data through an actual camera, the electronic device 1000according to the present disclosure collects data using the cameracharacteristic model 1510 and the 3D CAD model 1511 in a virtualenvironment. Accordingly, it is possible to collect data on a desiredmeasurement target object.

Also, the electronic device 1000 according to the present disclosure cangenerate training data which is robust to various variable surroundingelements by randomly changing various types of domain information.According to an exemplary embodiment, domain information is textureinformation and may include information on how light is reflected by anobject. According to an exemplary embodiment, domain information mayvary according to a path along which light is reflected by an object.

The electronic device 1000 according to the present disclosure may applythe 3D CAD model 1511 and the camera characteristic model 1510 into avirtual environment and then image a plurality of measurement targetobjects shown in the 3D CAD model 1511 through the camera characteristicmodel 1510. In particular, every time a 3D CAD model is imaged in avirtual environment through a camera characteristic model, theelectronic device 1000 according to the present disclosure can generatetraining data robust to various variable surrounding elements byrandomly changing domain information. The electronic device 1000 maytrain a first neural network model and a second neural network model onthe basis of training data generated according to the above-describedmethod.

FIG. 16 is a block diagram of an electronic device according to anexemplary embodiment.

FIG. 17 is a block diagram of an electronic device according to anotherexemplary embodiment.

As shown in FIG. 16, the electronic device 1000 according to theexemplary embodiment may include a processor 1300 and a memory 1700.However, all elements shown in the drawing are not essential. Theelectronic device 1000 may be implemented as more or fewer elements thanthe elements shown in the drawing.

For example, as shown in FIG. 17, the electronic device 1000 may includea user input interface 1100, an output unit 1200, a sensing unit 1400, anetwork interface 1500, an audio/video (A/V) input unit 1600, and amemory 1700 in addition to the processor 1300 and the memory 1700.

The user input interface 1100 is a means for a user to input data forcontrolling the electronic device 1000. For example, the user input unit1100 may be a key pad, a dome switch, a touch pad (a capacitive overlaytype, a resistive overlay type, an infrared beam type, a surfaceacoustic wave type, an integral strain gauge type, a piezoelectric type,etc.), a jog wheel, a jog switch, or the like, but the user input unit1100 is not limited thereto. The user input interface 1100 may receiveat least one user input for the electronic device 1000 to identifyinformation on a measurement target object.

The output unit 1200 may output an audio signal, a video signal, or avibration signal and include a display unit 1210, a sound output unit1220, and a vibration motor 1230. The display unit 1210 includes ascreen for displaying or outputting information processed by theelectronic device 1000. Also, the screen may display an image. Forexample, at least a part of the screen may display a target object imageacquired by imaging the measurement target object and a reference objectimage acquired by imaging a reference object.

The sound output unit 1220 outputs audio data received from the networkinterface 1500 or stored in the memory 1700. Also, the sound output unit1220 outputs a sound signal related to a function (e.g., a call signalring tone, a message ring tone, and a notification tone) performed bythe electronic device 1000. The processor 1300 generally controlsoverall operations of the electronic device 1000. For example, theprocessor 1300 may control the user input interface 1100, the outputunit 1200, the sensing unit 1400, the network interface 1500, the A/Vinput unit 1600, etc. overall by executing programs stored in the memory1700. Also, the processor 1300 may perform the functions of theelectronic device 1000 illustrated in FIGS. 1 to 15 by executing theprograms stored in the memory 1700.

According to an exemplary embodiment, the processor 1300 may image thereference object, which is a reference for identifying size of themeasurement target object, to acquire the reference object image, imagethe measurement target object to acquire the target object image, fusethe acquired reference object image and target object image, and inputthe fused reference object image and target object image to a firstneural network model to acquire size information of the measurementtarget object from the first neural network model.

According to an exemplary embodiment, the processor 1300 may acquire atarget object mask image, which is generated on the basis of a targetobject area in the target object image, from a second neural networkmodel by inputting the target object image to the second neural networkmodel and fuse the target object mask image and the reference objectimage.

According to an exemplary embodiment, the processor 1300 may acquire areference object feature from the second neural network model byinputting the reference object image to the second neural network modeland acquire size information of the measurement target object byinputting the acquired reference object feature and the acquired targetobject image to the first neural network model.

The sensing unit 1400 may sense a state of the electronic device 1000 ora state of surroundings of the electronic device 1000 and transfersensing information to the processor 1300. The sensing unit 1400 may beused to generate some of specification information of the electronicdevice 1000, state information of the electronic device 1000,surrounding information of the electronic device 1000, state informationof the user, and the user's history of using the electronic device 1000.

The sensing unit 1400 may include at least one of a magnetic sensor1410, an acceleration sensor 1420, a temperature/humidity sensor 1430,an infrared sensor 1440, a gyroscope sensor 1450, a location sensor(e.g., Global Positioning System (GPS)) 1460, an atmospheric sensor1470, a proximity sensor 1480, and a red green blue (RGB) sensor (i.e.,illuminance sensor) 1490, but the sensing unit 1400 is not limitedthereto. The function of each of the sensors may be intuitively derivedfrom the name by those of ordinary skill in the art, and a detaileddescription thereof is omitted.

The network interface 1500 may include at least one element which allowsthe electronic device 1000 to communicate with another device (notshown) and the server 2000. The other device (not shown) may be acomputing device like the electronic device 1000 or a sensing device butis not limited thereto. For example, the network interface 1500 mayinclude a short-range wireless communication unit 1510, a mobilecommunication unit 1520, and a broadcast receiving unit 1530.

The short-range wireless communication unit 1510 may include a Bluetoothcommunication unit, a Bluetooth Low Energy (BLE) communication unit, anear field communication (NFC) unit, a wireless local area network(WLAN) (i.e., Wi-Fi) communication unit, a ZigBee communication unit, aninfrared data association (IrDA) communication unit, a Wi-Fi direct(WFD) communication unit, an ultra-wideband (UWB) communication unit,etc., but the short-range wireless communication unit 1510 is notlimited thereto.

The mobile communication unit 1520 exchanges wireless signals with atleast one of a base station, an external terminal, and a server in amobile communication network. Here, the wireless signals may includevarious forms of data according to transmission and reception of voicecall signals, video call signals, or text or multimedia messages.

The broadcast receiving unit 1530 externally receives a broadcast signaland/or broadcast-related information through a broadcast channel Thebroadcast channel may include a satellite channel and a terrestrialchannel. According to an implementation example, the electronic device1000 may not include the broadcast receiving unit 1530. Also, thenetwork interface 1500 may acquire the target object image or thereference object image from an external device including a camera deviceor the server 2000. According to an exemplary embodiment, the networkinterface 1500 may transmit information on the measurement target objectidentified by the electronic device 1000 to the server 2000 or anexternal device.

The A/V input unit 1600 is intended to input an audio signal or a videosignal and may include a camera module 1610, a microphone 1620, and thelike. The camera module 1610 may acquire a video frame, such as a stillimage or a video, through an image sensor in a video call mode or animaging mode. An image captured through the image sensor may beprocessed through the processor 1300 or an additional image processingunit (not shown).

The microphone 1620 receives and processes an external sound signal intoelectrical voice data. For example, the microphone 1620 may receive asound signal from an external device or the user. The microphone 1620may receive a voice input of the user. The microphone 1620 may usevarious noise removal algorithms for removing noise which occurs in aprocess of receiving an external sound signal.

The memory 1700 may store a program for processing and control of theprocessor 1300 and data which is input to or output from the electronicdevice 1000. Also, the memory 1700 may store a result of searching forthe target object image, the reference object image, and images storedin the memory 1700.

Also, the memory 1700 may store information on at least one neuralnetwork model used by the electronic device 1000. For example, thememory 1700 may store weight values for layers and nodes in the at leastone neural network model and connection strength between the layers. Theelectronic device 1000 may further store training data which isgenerated by the electronic device 1000 to train a neural network model.Also, the memory 1700 may further store a 3D CAD model and a cameracharacteristic model for generating training data.

The memory 1700 may include at least one type of storage medium among aflash memory type memory, hard disk type memory, a multimedia card microtype memory, a card type memory (e.g., a Secure Digital (SD) memory oran eXtreme Digital (XD) memory), a random access memory (RAM), a staticRAM (SRAM), a read-only memory (ROM), an electrically erasableprogrammable ROM (EEPROM), a programmable ROM (PROM), a magnetic memory,a magnetic disk, and an optical disk.

The programs stored in the memory 1700 may be classified into aplurality of modules by function. For example, the programs may beclassified into a user interface (UI) module 1710, a touch screen module1720, a notification module 1730, and the like.

The UI module 1710 may provide a specialized UI, a graphics userinterface (GUI), etc. interoperating with the electronic device 1000according to applications. The touch screen module 1720 may detect theuser's touch gesture on a touch screen and transfer information on thetouch gesture to the processor 1300. The touch screen module 1720according to some exemplary embodiments may recognize and analyze atouch code. The touch screen module 1720 may be configured as separatehardware including a controller.

The notification module 1730 may generate a signal for notifying a userof the occurrence of an event. Examples of an event which occurs in theelectronic device 1000 are call signal receiving, message receiving, keysignal inputting, schedule notification, and the like. The notificationmodule 1730 may output a notification signal in the form of a videosignal through the display unit 1210, output a notification signal inthe form of an audio signal through the sound output unit 1220, andoutput a notification signal in the form of a vibration signal throughthe vibration motor 1230.

According to an exemplary embodiment, it is possible to effectivelyidentify size of a measurement target object.

According to an exemplary embodiment, it is possible to simply identifysize of a measurement target object on the basis of an estimatedposition of a camera without imaging a reference object together.

The method of identifying size of a measurement target object by anelectronic device according to an exemplary embodiment may beimplemented in the form of a program instruction executable by variouscomputing units and recorded on a computer-readable recording medium.The computer-readable recording medium may include program commands,data files, data structures, etc. alone or in combination. The programcommands recorded on the medium may be those specially designed andconfigured for the present disclosure or may be those publicly known andavailable to those of ordinary skill in the field of computer software.

Examples of the computer-readable recording medium include magneticmedia, such as hard disks, floppy disks, and magnetic tapes, opticalrecording media, such as compact disc (CD)-ROMs and digital versatilediscs (DVDs), magneto-optical media, such as floptical disks, andhardware devices such as ROMs, RAMs, and flash memories speciallyconfigured to store and execute program commands Examples of programcommands include machine language codes made by a compiler andhigh-level language codes executable by a computer using an interpreterand the like.

Some of the exemplary embodiments may be implemented in the form of arecording medium which includes instructions executable by a computersuch as program modules executed by computers. The computer-readablemedium may include any available medium that may be accessed bycomputers and may include all volatile and non-volatile media anddetachable and non-detachable media. Also, the computer-readable mediummay include both a computer storage medium and a communication medium.The computer storage medium may include all volatile and non-volatilemedia and detachable and non-detachable media which are based on anymethods or technologies for storing information includingcomputer-readable instructions, data structure, program modules, orother data. The communication medium may typically includecomputer-readable instructions, data structures, program modules, otherdata of modified data signals, such as carrier waves, other transmissionmechanisms, or any information transmission media. Also, some of theexemplary embodiments may be implemented as a computer program orcomputer program product including instructions executable by acomputer.

Although the exemplary embodiments of the present disclosure have beendescribed in detail, the scope of the present disclosure is not limitedthereto, and various modifications and alterations made by those ofordinary skill in the art using the basic concept defined in thefollowing claims fall within the scope of the present disclosure.

What is claimed is:
 1. A method of identifying a size of a measurementtarget object by an electronic device, the method comprising: imaging areference object, which is a reference for identifying the size of themeasurement target object, to acquire a reference object image; imagingthe measurement target object to acquire a target object image; fusingthe acquired reference object image and the acquired target objectimage; and inputting the fused reference object image and target objectimage to a first neural network model to acquire size information of themeasurement target object from the first neural network model.
 2. Themethod of claim 1, further comprising inputting the target object imageto a second neural network model to acquire a target object mask imagegenerated on the basis of a target object area in the target objectimage from the second neural network model, wherein the fusing of theacquired reference object image and target object image comprises fusingthe target object mask image and the reference object image.
 3. Themethod of claim 1, further comprising: inputting the reference objectimage to a second neural network model to acquire a reference objectfeature from the second neural network model; and inputting the acquiredreference object feature and the acquired target object image to thefirst neural network model to acquire size information of themeasurement target object.
 4. The method of claim 1, further comprising:inputting the reference object image to a second neural network model toacquire a reference object feature from the second neural network model;inputting the target object image to the second neural network model toacquire a target object feature from the second neural network model;fusing the acquired reference object feature and the acquired targetobject feature; and inputting the fused reference object feature andtarget object feature to the first neural network model to acquire sizeinformation of the measurement target object.
 5. The method of claim 1,further comprising: inputting the reference object image to a secondneural network model to acquire an output value of the second neuralnetwork model; inputting the target object image to the second neuralnetwork model to acquire an output value of the second neural networkmodel; fusing the acquired output values of the second neural networkmodel; and inputting the fused output values of the second neuralnetwork model to the first neural network model to acquire sizeinformation of the measurement target object.
 6. The method of claim 1,wherein the fusing of the acquired reference object image and theacquired target object image comprises applying information on thereference object image to the target object image to fuse the referenceobject image and the target object image or fusing the reference objectimage and the target object image in units of certain channels.
 7. Themethod of claim 2, wherein the second neural network model includes anobject detection network configured to identify an object area includingeach of the at least one measurement target object in the target objectimage, which is input to the second neural network model and includesthe at least one measurement target object, generate an object boxincluding the object area, identify a type of the measurement targetobject represented by the object box, and binarize an image input to thesecond neural network model on the basis of the object area to generatethe target object mask image.
 8. The method of claim 7, wherein thefirst neural network model includes an object size identificationnetwork configured to acquire the target object mask image and thereference object image, compare the acquired target object mask imageand the acquired reference object image, and output information on thetype and size of the measurement target object, which is represented bythe object box in the target object image including the measurementtarget object, on the basis of a comparison result.
 9. The method ofclaim 2, wherein the first neural network model and the second neuralnetwork model are trained in advance on the basis of training data whichis generated using a preset computer aided design (CAD) model and acamera characteristic model including a preset camera characteristicparameter in a virtual environment for virtually imaging the measurementtarget object and the reference object.
 10. The method of claim 9,wherein the training data is generated using the CAD model and thecamera characteristic model while changing domain information whichvaries on the basis of a path along which virtual light is reflected bythe measurement target object or the reference object.
 11. The method ofclaim 1, further comprising acquiring information on a type of themeasurement target object and an area of the measurement target objectin the target object image from the first neural network model.
 12. Anelectronic device for identifying a size of a measurement target object,the electronic device comprising: a memory configured to store one ormore instructions; and at least one processor configured to execute theone or more instructions, wherein the at least one processor executesthe one or more instructions so that a reference object, which is areference for identifying the size of the measurement target object, isimaged to acquire a reference object image, the measurement targetobject is imaged to acquire a target object image, the acquiredreference object image and the acquired target object image are fused,and the fused reference object image and target object image is input toa first neural network model to acquire size information of themeasurement target object from the first neural network model.
 13. Theelectronic device of claim 12, wherein the at least one processor inputsthe target object image to a second neural network model to acquire atarget object mask image generated on the basis of a target object areain the target object image from the second neural network model andfuses the target object mask image and the reference object image. 14.The electronic device of claim 12, wherein the at least one processorinputs the reference object image to a second neural network model toacquire a reference object feature from the second neural network modeland inputs the acquired reference object feature and the acquired targetobject image to the first neural network model to acquire sizeinformation of the measurement target object.
 15. A computer-readablerecording medium in which a program for performing a method ofidentifying a size of a measurement target object by an electronicdevice is stored, wherein the method comprises: imaging a referenceobject, which is a reference for identifying the size of the measurementtarget object, to acquire a reference object image; imaging themeasurement target object to acquire a target object image; fusing theacquired reference object image and the acquired target object image;and inputting the fused reference object image and target object imageto a first neural network model to acquire size information of themeasurement target object from the first neural network model.